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(54) 3d image acquisition apparatus and 3d image acquisition method 



(57) A 3D image acquisition apparatus comprises a 
pattern projection section (1 02) which projects a prede- 
termined pattern on an object to be measured, an imag- 
ing section (103, 104) which is disposed at a predeter- 
mined distance from the pattern projection section and 
images the object on which the predetermined pattern 
has been projected, and a depth calculation section 
(106) which detects the projection pattern projected on 
the object on the basis of an image acquired by the im- 
aging section, collates the detected pattern and the pre- 
determined pattern projected by the pattern projection 



section, and calculates a depth of respective parts of the 
object on the basis of the correspondency of the colla- 
tion. The predetermined pattern projected by the pattern 
projection section is a pattern (e.g. stripes or matrix) 
formed by alternately arranging areas with local maxi- 
mum luminance values and areas with local minimum 
luminance values. Thus, stripes/matrix boundaries 
(blocks) can be exactly extracted from the pattern pro- 
jection image, and correct decoding is performed from 
the encoded projection image even where the object is 
not a white-based color one or a low-saturation color 
one. 
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Description 

[0001] The present invention relates to a 3D (Three 
Dimensional) image acquisition method for a technique 
of measuring a 3D shape of an object in a non-contact 
state, and a 3D image acquisition apparatus for carrying 
out this method. 

[0002] Conventional 3D object shape measuring 
methods applied to 3D image acquisition apparatuses 
as 3D shape measuring techniques are generally clas- 
sified into two types. One is a method in which the time 
of propagation of light is measured, and the other is a 
method making use of a "principle of triangulation". The 
former is a theoretically ideal method because there is 
no dead angle, but it has problems with measured time 
and precision. Thus, the latter using "principle of trian- 
gulation" is mainly used. 

[0003] Methods using the principle of triangulation are 
classified into an active method and a passive method 
called "passive stereo method". In the passive stereo 
method, features in the images obtained by two camer- 
as situated at different locations are made to corre- 
spond. On the basis of a result of the correspondence 
and a p re-measured positional relationship between two 
cameras, a distance up to an object is found by the prin- 
ciple of triangulation. The passive stereo method, how- 
ever, has drawbacks that the correspondence of fea- 
tures in the images is difficult and the shape of an object 
without texture cannot be measured. 
[0004] As an example of the active method using the 
principle of triangulation, there is known a "light projec- 
tion method". In this method, one of the two cameras, 
is replaced with a light source, and an image of the light 
source at the surface of the object is observed by the 
camera situated at the other point and the shape of the 
object is measured. The "light projection method" is also 
classified into "spot light projection method", "slit light 
projection method" and "pattern light projection meth- 
od". 

[0005] In the spot light projection method, only a point 
of an object can be measured by a single image input 
In the slit light projection method, one line of an object 
can be measured by a single image input. However, in 
order to measure the shape of the entire area of the ob- 
ject, image input needs to be repeated a number of 
times by scanning projection light, and a great deal of 
time is required. 

[0006] On the other hand, in the pattern light projec- 
tion method, a two-dimensional (2D) pattern such as a 
stripe pattern or a matrix pattern is projected. This meth- 
od is advantageous in that only one input of a pattern 
projection image needs to be performed. 
[0007] The pattern light projection method is also 
called "spatial encoding method". The "spatial encoding 
method" is classified into "pattern shape encoding" and 
"gradation encoding". As the pattern shape encoding, a 
method using a distribution of opening width of a slit and 
a method using an M-series encoding have been pro- 



posed, but these are less practical because of problems 
with measurement density and stability of measure- 
ment. 

[0008] Meanwhile, the "gradation encoding" is 
s classed into "gray scale encoding" and "color tone en- 
coding". However, either method has a drawback that 
the number of recognizable gradations is small. As re- 
gards the gray scale encoding, the maximum number of 
recognizable gradations is normally 5. As regards the 
10 color tone encoding, a greater number of gradations are 
not usable, except the cases of white and tow-saturation 
colors. 

[0009] To solve these problems, a method has been 
proposed wherein one code is assigned to a combina- 

15 tion a plurality of colors or gray levels. 

[0010] Prior-art matrix pattern encoding and stripe 
pattern encoding will now be described. 
[0011] As regards the matrix pattern, Patent No. 
2,565,885, for instance, discloses a "spatial pattern en- 

20 coding method". This method employs a multi-pattern 
matrix plate pattern, which has gray scale areas of three 
or more values, color areas of three or more colors, or 
three or more gradation areas of a combination of gray 
scales and colors. In this method, at least three kinds of 

25 gradation areas adjoin at an intersection of boundary 
lines of gradation areas. Major codes corresponding to 
the kinds and order of gradations are assigned to inter- 
sections of a projection image produced by projecting 
the pattern on the object. Then, the major codes, orcom- 

30 binational codes of major code at the intersection and 
major codes of intersections around this intersection, 
are given as characteristic codes for intersection recog- 
nition. 

[001 2] FIG . 9 shows an example of the matrix pattern 
35 using three-value gradations in the above-described pri- 
or-art spatial pattern encoding method. White is repre- 
sented by "2", black is by "0", and gray is by "1". FIG. 
10 is a block flow chart illustrating the process by the 
spatial pattern encoding method. 
40 [0013] In step ST1001, recognition thresholds for 
three-value conversion are found. In step ST1002, 
three-value conversion for each pixel is effected, like 
"0", "1"or"2". 

[0014] In step ST1003, node selection is effected to 
*5 extract, as a node, the comer or edge intersection of 
each matrix plate area of the multi-value matrix pattern. 
[0015] In step ST1004, a decoding process is per- 
formed to provide one of a plurality of codes as a major 
code in accordance with a gradation distribution sur- 
so rounding the extracted node. 

[0016] In step ST1005, a result of code recognition 
and code numbering is recorded on a table (memory). 
[0017] This method is advantageous in that it provides 
a pattern structure which can produce many character- 
's tstic codes by only single projection of the pattern with 
at least three gradations. 

[0018] In order to derive encoded information from a 
received pattern and derive 3D information, it is impor- 



2 



EP 1 217 328 A1 



4 



tant to exactly read a luminance variation and to detect 
as exactly as possible boundaries of one block in the 
matrix. A graph shown below the matrix pattern in FIG. 
9 indicates gradations of the second line from above in 
the matrix pattern. The inclination of a graph curve in- 
creases near inflection points (i.e. two occurrences of 
(1) in (1)(2)(1) in FIG. 9), which are neither local maxi- 
mum points or local minimum points, due to diffusion of 
projected light. Consequently, extraction of a matrix 
structure (division of blocks in the matrix) becomes dis- 
advantageously unstable. In short, the degree of devia- 
tion to the left and right of an inflection point increases, 
and the reliability lowers. 

[0019] FIG. 11 shows prior art in the case where gra- 
dations of three colors, R (Red), G (Green) and B (Blue), 
are used in the spatial pattern encoding method. As- 
sume that "0" is assigned to R, "1" to G, and "2" to B. 
[0020] A graph below the matrix pattern in FIG. 11 in- 
dicates luminance values of R, G and B of the second 
line from above in the matrix pattern. For example, in 
the case of an object (a high-saturation red object) hav- 
ing such surface reflectance characteristics that reflec- 
tion signals of a G component and a B component are 
substantially zero while a reflection signal of an R com- 
ponent is large, a signal of a local maximum is obtained 
at the luminance value of R in the second line from 
above in the pattern of FIG. 11. By contrast, the lumi- 
nance values of G and B have neariy noise levels. In 
this situation, decoding is impossible. Similarly, decod- 
ing is very difficult for an object having such surface re- 
flectance characteristics that even one of reflection sig- 
nals of R, G and B is substantially zero. In order words, 
decoding is successfully carried out for an object sur- 
face of a white-based color or a low-saturation color. 
However, for an object having such surface reflectance 
characteristics that even one of reflection signals of R, 
G and B components may take a nearly zero value, de- 
termination based on a color obtained from a combina- 
tion of R, G and B is difficult 

[0021] Prior art of a stripe pattern, which is disclosed 
in, e.g. the Journal of the Institute of Electronics and 
Communication Engineers (Vol. J61-D, No. 6, pp. 
411-418, 1978), will now be described. FIG. 12 shows 
an example of a stripe pattern encoded by colors of R, 
G and B. This example differs from the example of FIG. 
11, not only because the former is a stripe and the latter 
is a matrix, but also because black areas (hatched in 
FIG. 1 2) are present among R, G and B. The presence 
of black areas reduces the possibility of mixing of colors 
due to diffusion of tight. 

[0022] However, in this prior-art example, too, decod- 
ing is difficult for an object having such surface reflect- 
ance characteristics that even one of reflection signals 
of R, G and B components may take a nearly zero value. 
[0023] The present invention has been made to solve 
the problems of the above-described prior art The ob- 
ject of this invention is to provide a 3D image acquisition 
method and a 3D image acquisition apparatus based on 



spatial pattern encoding, wherein stripe boundary ex- 
traction from a stripe pattern projection image and block 
boundary extraction from a matrix pattern projection im- 
age can exactly be performed, and exact decoding is 
5 performed from an encoded projection image of an ob- 
ject even where the object is not of a white-based color 
or a low-saturation color. 

[0024] In order to solve the above problems and 
achieve the object, the present invention provides the 

10 following means. Specifically, in the 3D image acquisi- 
tion method of this invention, spatial pattern encoding is 
used to measure the shape of a 3D object in a non-con- 
tact state. A pattern is projected, and the corresponden- 
cy between a reception-light pattern and a projection- 

'5 light pattern is found. Thereby, depth information of an 
object is acquired. The 3D image acquisition apparatus 
of this invention is provided to carry out this method. 
[0025] According to a first aspect of the invention, 
there is provided a 3D image acquisition apparatus com- 

20 prising: a pattern projection section which projects a pre- 
determined pattern on an object; an imaging section 
which is disposed at a predetermined distance from the 
pattern projection section and images the object on 
which the predetermined pattern has been projected; 

25 and a depth calculation section which detects the pro- 
jection pattern projected on the object on the basis of 
an image acquired by the imaging section, collates the 
detected projection pattern and the predetermined pat- 
tern projected by the pattern projection section, and cal- 

30 culates a depth of respective parts of the object on the 
basis of the correspondency of the collation, character- 
ized in that the predetermined pattern projected by the 
pattern projection section is formed by alternately ar- 
ranging areas with local maximum luminance values 

35 and areas with local minimum luminance values. 

[0026] According to a second aspect of the invention, 
there is provided a 3D image acquisition apparatus com- 
prising: a pattern projection section which projects a pre- 
determined pattern on an object; an imaging section 
which is disposed at a predetermined distance from the 
pattern projection section and images the object on 
which the predetermined pattern has been projected; 
and a depth calculation section which detects an image 
pattern on the basis of an image acquired by the imaging 

45 section, collates the detected image pattern and the pre- 
determined pattern projected by the pattern projection 
section, and calculates a depth of respective parts of the 
object on the basis of the correspondency of the colla- 
tion, characterized in that the predetermined pattern 

so projected by the pattern projection section is formed by 
combining patterns of a plurality of color components, 
and the patterns of the plurality of color components are 
formed by alternately arranging areas with local maxi- 
mum luminance values and areas with local minimum 

55 luminance values. 

[0027] In this 3D image acquisition apparatus, in the 
patterns of the plurality of color components, positions 
of the areas with local maximum luminance values and 



3 



EP1 217328 A1 



6 



positions of the areas with local minimum luminance val- 
ues are aligned. 

[0028] In this 3D image acquisition apparatus, in the 
patterns of the plurality of color components, positions 
of the areas with local maximum luminance values and 
positions of the areas with local minimum luminance val- 
ues are shifted. 

[0029] In the above 3D image acquisition apparatus, 
the color components are RGB components. 
[0030] In this 3D image acquisition apparatus, the 
projected predetermined pattern is a stripe pattern or a 
matrix pattern. 

[0031] According to a third aspect of the invention, 
there is provided a 3D image acquisition method com- 
prising: a step of projecting a predetermined pattern on 
an object; a step of imaging the object on which the pre- 
determined pattern has been projected, at a position at 
a predetermined distance from a position where the pre- 
determined pattern has been projected; and a step of 
detecting a pattern on the basis of an image acquired 
by the imaging step, collating the detected pattern and 
the pattern projected by the pattern projection step, and 
calculating a depth of respective parts of the object on 
the basis of the correspondency of the collation, char- 
acterized in that the predetermined pattern projected by 
the pattern projection step is formed by alternately ar- 
ranging areas with local maximum luminance values 
and areas with local minimum luminance values. 
[0032] According to a fourth aspect of the invention, 
there is provided a 3D image acquisition method com- 
prising: a step of projecting a predetermined pattern on 
an object; a step of imaging the object on which the pre- 
determined pattern has been projected, at a position at 
a predetermined distance from a position where the pre- 
determined pattern has been projected; and a step of 
detecting a pattern on the basis of an image acquired 
by the imaging step, collating the detected pattern and 
the pattern projected by the pattern projection step, and 
calculating a depth of respective parts of the object on 
the basis of the correspondency of the collation, char- 
acterized in that the predetermined pattern projected by 
the pattern projection step is formed by combining pat- 
terns of a plurality of color components, and the patterns 
of the plurality of color components are formed by alter- 
nately arranging areas with local maximum luminance 
values and areas with local minimum luminance values. 
[0033] This summary of the invention does not nec- 
essarily describe ail necessary features so that the in- 
vention may also be a sub-combination of these de- 
scribed features. 

[0034] The invention can be more fully under stood 
from the following detailed description when taken in 
conjunction with the accompanying drawings, in which: 

FIG. 1 illustrates a light pattern structure when pat- 
tern light is projected in a 3D image acquisition 
method according to a first embodiment of the 
present invention; 



FIG. 2 schematically shows the structure of a 3D 
image acquisition apparatus according to the first 
embodiment; 

FIG. 3 is a block flow chart illustrating the process 
5 in the 3D image acquisition method according to the 
first embodiment; 

FIG. 4 schematically shows the structure of a 3D 
image acquisition apparatus according to a second 
embodiment of the invention; 
to FIG. 5 illustrates a light pattern structure when pat- 
tern light is projected in a 3D image acquisition 
method according to a third embodiment of the in- 
vention; 

FIG. 6 is a block flow chart illustrating the process 
« in the 3D image acquisition method according to the 
third embodiment; 

FIG. 7 illustrates a light pattern structure when pat- 
tern light is projected in a 3D image acquisition 
method according to a fourth embodiment of the in- 
20 vention; 

FIG. 8 illustrates a light pattern structure when pat- 
tern light is projected in a 3D image acquisition 
method according to a fifth embodiment of the in- 
vention; 

25 FIG. 9 shows an example of a matrix pattern using 
three-value gradations in a prior-art spatial pattern 
encoding method; 

FIG. 1 0 is a block flow chart illustrating the process 
in the prior-art spatial pattern encoding method; 
30 FIG. 1 1 shows an encoded light pattern structure in 
prior-art matrix pattern light projection; and 
FIG. 12 shows a light pattern structure in prior-art 
matrix pattern light projection. 

3s [0035] Embodiments of the present invention will now 
be described with reference to FIGS. 1-8. 

(First Embodiment) 

40 [0036] A first embodiment of the invention will now be 
described in detail with reference to FIGS. 1-3. 
[0037] In order to describe a pattern light projection 
method for spatial pattern encoding according to the first 
embodiment, FIG. 1 shows a four-gradation stripe pat- 

45 tern byway of example. This pattern is a predetermined 
light pattern which is projected on an object when a 3D 
image of the object is to be acquired, for example, in 
order to measure the shape of the object 
[0038] This predetermined pattern comprises a plu- 

50 rality of areas (stripes in FIG. 1 ) each having a substan- 
tially constant luminance value. The luminance (grada- 
tion) of each area belongs to any one of predetermined 
luminances (gradations). In FIG. 1 , each stripe belongs 
to any one of four gradations (0, 1 , 2, 3). In this example, 

55 pattern areas having local maximum luminance values 
and local minimum luminance values are alternately ar- 
ranged. 

[0039] Each area is arranged such that if one of its 
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adjacent areas has a higher luminance (gradation) than 
itself, the other adjacent area also has a higher lumi- 
nance (gradation) than itself. In addition, each area is 
arranged such that if one of its adjacent areas has a low- 
er luminance (gradation) than itself, the other adjacent 
area also has a lower luminance (gradation) than itself. 
Both adjacent areas have different luminances (grada- 
tions). 

[0040] In the encoding of respective stripe patterns, 
as shown in the graph of FIG. 1 , local maximum values 
and local minimum values are alternately arranged. Lo- 
cal maximum values of gradations 1 , 2 and 3 are ex- 
pressed by (1), (2) and (3), and local minimum values 
of gradations 0, 1 and 2 are expressed by (0), (-1) and 
(-2). Using these expressions, a description wilt be given 
of the spatial pattern encoding method in which the 
same code does not appear twice in the arrangement 
of the stripes. In this case, the order of stripe patterns 
are determined under conditions that the gradation of a 
location minimum is lower than the gradations of right 
and left local maximums, odd-number-th stripes have 
local maximum values, and even-number-th stripes 
have local minimum values. 

[0041] In the example of FIG. 1 , if four numerals ar- 
ranged in the order of a local maximum, a local mini- 
mum, a local maximum and a local minimum are encod- 
ed into one code, two codes, i.e. [(1 )(0)(2)(-1 )] and [(2) 
(-1)(3)(-2)], are obtained. In this case, the two succes- 
sive codes share numerals (2) and (-1), which are the 
right-hand two numerals of the left-hand code and are 
the left-hand two numerals of the right-hand code. If 
codes are arranged to share the same numerals, ar- 
rangements of (2m+2) stripes are obtained when the 
number of codes is m. When the number of gradations 
is 4 and the code length is 4, the number of codes, m, 
is 30. Accordingly, in the case where information of 4 
gradations is combined with local maximum/minimum 
information, if encoding is effected by the arrangement 
of four stripes, 62 stripes are obtained. 
[0042] The spatial pattern encoding process accord- 
ing to the present invention will now be described with 
reference to a flow chart of FIG. 3. In step ST301, stripes 
at positions of local maximums and local minimums are 
detected on the basis of a pattern projection image. 
These stripes can be detected by extracting edge por- 
tions of stripes through a "Sober fitter or the like and 
determining whether an area interposed between the 
detected edge portions has a local maximum or a local 
minimum. 

[0043] In the next step ST302, the gradation of the 
area interposed between the edge portions is deter- 
mined. 

[0044] In step ST303, a decoding process is per- 
formed by numerically converting the information of lo- 
cal maximum/minimum of the area between edge por- 
tions and the gradation information. In step ST304, the 
obtained numerical information is recorded on a table in 
a memory. The numerical information is collated with 



prestored information in a pattern memory 101 (to be 
described later). 

[0045] In the case of four gradations, a code having 
the coincident order of arrangement of four numerals is 
5 found. If there are two or more coincident codes, a code 
having a greatest length of a successive coincident por- 
tion is selected as one with highest reliability. In this way, 
the correspondency of 1 to 1 (at most) is determined 
between each stripe of a projection-light pattern and 
each stripe of a reception-light pattern, and depth-di- 
mensional information of the object to be measured is 
calculated from the well-known "principle of tri angula- 
tion*. 

[0046] A description will now be given of a 3D image 
acquisition apparatus for acquiring a 3D image by using 
a pattern obtained by the above-described spatial pat- 
tern encoding method. 

[0047] FIG. 2 schematically shows an example of the 
structure of a 3D image acquisition apparatus according 
to the first embodiment. This 3D image acquisition ap- 
paratus projects a predetermined pattern on an object 
to be measured, acquires an image of the object, and 
collating an obtained detection pattern and the projected 
pattern, thereby measuring depth dimensions of respec- 
tive parts of the object. This apparatus comprises the 
following elements. 

[0048] A pattern memory 1 01 that prestores a prede- 
termined pattern of projection light is connected to a pat- 
tern projection section 1 02 for projecting the light of the 
predetermined pattern on an object to be measured (not 
shown). This predetermined pattern is projected from 
the pattern projection section 1 02 on the object in the 
state in which areas with local maximum luminance val- 
ues and local minimum luminance values are alternately 
arranged, for example, as shown in FIG. 2. 
[0049] Imaging means for imaging the object, on 
which the predetermined pattern has been projected, is 
disposed at a predetermined distance from the pattern 
projection section 1 02. The imaging means comprises 
a lens 1 03 for collecting reflection light from the object, 
and a light reception section 104 for receiving the col- 
lected reflection light. 

[0050] An image memory 1 05 that stores information 
of the light received by the light reception section 104 
of the imaging means is connected to the imaging 
means. The image memory 105 and pattern memory 
101 are connected so that a pattern can be detected on 
the basis of an image acquired by the imaging means 
(the imaging section including the light collection lens 
1 03 and light reception section 104). A depth calculation 
section 106 is provided as depth calculation means for 
collating the detected pattern with the pattern projected 
by the pattern projection section 102, and calculating the 
depth dimension of the respective parts of the object on 
the basis of the correspondency between the two col- 
lated patterns. 

[0051] A 2D image information generating section 
1 07 for generating 2D image information on the basis of 
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the information stored in the image memory 105 is con- 
nected to a 3D image generating section 108. The 3D 
image generating section 108 generates a 3D image on 
the basis of the 2D image information generated by the 
2D image information generating section 107 and the 
depth dimension value calculated by the depth calcula- 
tion section 106. 

[0052] The operation of the 3D image acquisition ap- 
paratus will now be described. 
[0053] Pattern encoding information is prestored in 
the pattern memory 101 . To start with, in order to acquire 
an image without a pattern, the light collection lens 1 03 
collects reflection light from the object, and the light re- 
ception section 104 receives the collected reflection 
light. The information of the received light is stored in 
the image memory 105. The 2D image generating sec- 
tion 1 07 generates two-dimensional photographic infor- 
mation. Based on this information, surface reflectance 
characteristics of the object are obtained. The surface 
reflectance characteristics of the object obtained from 
the image with no pattern are used as a base (reference) 
for comparison of luminance, when pattern information 
is extracted from the image acquired from the object on 
which the pattern light is projected. 
[0054] A predetermined pattern is projected on the 
object, and reflection light from the object is collected 
by the collection light 1 03. The collected light is received 
by the light reception section 1 04 and then stored in the 
image memory 105. The depth calculation section 108 
compares and collates the information stored in the pat- 
tern memory 101 and the pattern information stored in 
the image memory 105, thereby determining the corre- 
spondency between the projection-light pattern and the 
reception-light pattern. 

[0055] The incidence angle is calculated on the basis 
of the information of the light reception point and the fo- 
cal distance, and the depth dimension is calculated by 
the principle of triangulation on the basis of the informa- 
tion of the distance between the light emission point and 
light reception point and the light emission angle. Since 
the optical parameters (e.g. focal distance) of the lens 
1 03 and the size of the light reception section are known, 
the incidence angle is determined on the basis of the 
position (light reception position) on the acquired image. 
The distance between the light emission point and light 
reception point is determined by the structure of the ap- 
paratus. The light emission angle is determined by the 
pattern on the image. 

[0058] At last, the 3D image generating section 1 08 
generates a 3D image on the basis of the 2D image gen- 
erated by the 2D image generating section 107 and the 
depth dimension value calculated by the depth calcula- 
tion section 106. 

[0057] According to the above-described structure of 
the 3D image acquisition apparatus of the first embodi- 
ment, the areas with local maximums and the areas with 
local minimums are alternately arranged on the pattern 
employed. Thus, the stripe image (stripe boundary) can 



easily be extracted. 

[0058] Moreover, each band of R, G and B is encoded 
based on the combination of local maximum/minimum 
information and gradation information. The range of ob- 
5 jects for 3D shape measurement is not limited to white- 
based color objects and low-saturation color objects and 
is increased. 
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[0059] Asecond embodiment of the present invention 
will now be described with reference to FIG. 4. The sec- 
ond embodiment is substantially similar to the first em- 
bodiment shown in FIG. 2 with respect to the spatial pat- 
's tern encoding method. However, as shown in FIG. 4, the 
structure of the 3D image acquisition apparatus is dif- 
ferent, as described below. The imaging means are pro- 
vided at a plurality of locations with different viewpoints, 
for example, on the right and left sides (two locations) 
2D of the pattern projection section 202. Accordingly, a pair 
of image memories are provided on the right and left 
sides. Moreover, means for determining the right-and- 
left correspondency is additionally provided. 
[0060] More specifically, the 3D image acquisition ap- 
25 paratus according to the second embodiment compris- 
es a pattern memory 201 storing a predetermined pro- 
jection-light pattern; a pattern projection section 202 for 
projecting the pattern light on the object; light collection 
lenses 203R and 203L and light reception sections 
30 204R and 204L constituting, e.g. right and left imaging 
means disposed at two locations; right and left image 
memories 205 R and 205L; a correspondency determi- 
nation section 206 for determining the correspondency 
between the projected pattern and detected pattern; a 
35 2D image information generating section 207; a depth 
calculation section 208; and a 3D image generating sec- 
tion 209. 

[0061 ] The operation of the 3D image acquisition ap- 
paratus according to the second embodiment will now 

40 be described. 

[0062] Like the first embodiment, images without pat- 
terns are acquired in advance. These images acquired 
at two locations with different viewpoints are stored in 
the image memories 205 R and 205L 

45 [Q063] If a predetermined pattern is projected on the 
object, the projected pattern is reflected by the object 
and collected by the collection lenses 203R and 203L 
disposed at two different positions (L (left) and R (right) 
in this case). The collected reflection light is received by 

50 the light reception sections 204 R and 204L, respective- 
ly, and then stored in the image memories 205R and 
2Q5L. 

[0064] The correspondency determination section 
206 performs four kinds of collations between:® the 
55 projection-light pattern and the pattern received by the 
left light reception section 204L;© the projection-light 
pattern and the right light reception section 204R; © 
the images without patterns received by the left light re- 
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ception section 204L and the right light reception section 
204R; and® the images with patterns received by the 
left light reception section 204L and the right light recep- 
tion section 204 R. This considerably reduces the diffi- 
culty in the "passive stereo method" in the collation with 
non-texture areas. 

[0065] In the first embodiment, the correspondency 
between the projection-light pattern and reception-light 
pattern is used as the basis for calculation of depth di- 
mensions. In the second embodiment, the correspond- 
ency between the right and left images without patterns 
and the correspondency between the right and left im- 
ages with patterns are also used, and the collation with 
higher reliability can be realized. 
[0066] At last, the 3D image generating section 209 
generates a 3D image on the basis of the 2D image in- 
formation obtained by the 2D image information gener- 
ating section 207 and the depth dimension value calcu- 
lated by the depth calculation section 208. 
[0067] As has been described above, according to the 
structure of the 3D image acquisition apparatus of the 
second embodiment, pattern projection images are ob- 
tained by the imaging elements situated at locations of 
two or more viewpoints. Not only the correspondency 
between the projection-light pattern and reception- light 
pattern but also the correspondency between plural im- 
ages is performed. Thus, more exact correspondency 
is achieved and the measurement precision and relia- 
bility are enhanced. 

(Third Embodiment) 

[0068] A third embodiment of the present invention 
will now be described with reference to FIGS. 5 and 6. 
[0069] The structure of the 3D image acquisition ap- 
paratus used in the first embodiment is substantially the 
same as that of the 3D image acquisition apparatus of 
the third embodiment However, the spatial pattern en- 
coding method is different, as shown in RG. 5. 
[0070] RG. 5 shows an example of a stripe pattern in 
which local maximums/minimums of a plurality of color 
signals are aligned. Gray scale values are used in the 
structure of the pattern adopted in the first embodiment 
On the other hand, the pattern used in the third embod- 
iment is a pattern of respective color components (R, G, 
B), and areas with local maximum luminance values and 
areas with local minimum luminance values are aligned. 
As is shown in RG. 5, in the color stripe pattern, local 
maximum positions and local minimum positions of 
RGB are aligned. 

[0071] R, G and B signals are individually encoded, 
and a decoding process is carried out by using R, G and 
B signals whose reflectances are not at noise level. If 
the reflectances of all R, G and B signals are not at noise 
level, a color decoding process is attempted, and color 
information is used in the decoding process. In addition, 
since local maximums and local minimums are alter- 
nately present, the extraction of a pattern image is ad- 



vantageously facilitated. 

[0072] A processing operation relating to the 3D im- 
age acquisition apparatus according to the third embod- 
iment will now be described. The flow chart of FIG. 6 
5 illustrates the process based on the 3D image acquisi- 
tion method. 

[0073] In step ST601 , image data without a pattern is 
input. 

[0074] In steps ST602R, ST602G and ST602B, the R, 

io G and B components of the image data are determined, 
and the amount of projected R, G and B light returned 
from the object is checked. Thereby, the reflectance 
characteristics of the surface of the object are approxi- 
mately determined. At this time, if the R component of 

*s the reflection light is at noise level and near zero, the R 
component of the pattern projection image is also near 
zero in the near region. Thus, the data of the R compo- 
nent cannot be decoded. The same applies to the data 
of the G component and B component. 

20 [0075] The above matters are determined. If the data 
of the R component of R, G and B of the image, for ex- 
ample, is not at noise level, the data of the R component 
of the pattern projection image is used (ST603R). At this 
time, in step ST604R, the local maximum/minimum of 

25 the R component data is detected. In subsequent step 
ST605R, the gradation of the local maximum/minimum 
area is found. Similarly, the data of the G component 
and B component is subjected to the same processing 
(ST603G, ST604G, ST605G; ST603B, ST604B, 

30 ST605B). 

[0076] On the other hand, in steps ST602T to ST605T, 
the following process is performed, if the surface reflect- 
ance characteristics of R, G and B are all above the 
noise level, the colors of the projection-light pattern are 

35 estimated. The estimation of colors is carried out by di- 
viding the RGB components of the pattern projection im- 
age by the RGB components of the image with no pat- 
tern and finding their ratio. If the surface reflectance 
characteristics of R, G and B are all higher than the noise 

40 level, the RGB data of the pattern projection image is 
used to detect local maximums/minimums of the data 
and to determine the colors. 

[0077] In step ST606, a decoding process is per- 
formed by numerically converting the information of lo- 

45 cal maximum/minimum of the area to be processed and 
the gradation information. In step ST607, the obtained 
numerical information is recorded on the table in the 
memory. Then, the numerical information is collated 
with prestored information in the pattern memory 101 . 

so Based on the principle of triangulation, the depth infor- 
mation of the object is calculated. 
[0078] In the pattern used in the 3D image acquisition 
apparatus of the third embodiment, the areas with local 
maximum luminance values and the areas with local 

55 minimum luminance values of R, G and B are aligned. 
The decoding can be performed on the basis of not only 
the luminance values and local maximum/minimum in- 
formation of R, G and B in the graph, but also the color 
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information determined by the R, G and B values in each 
stripe. 

[0079] Even where two of the three R, G and B signals 
are not at the noise level, the two signals can be used 
for the decoding. 

[0080] Since encoding for R, G and B is individually 
performed, the decoding process can be performed by 
combining the individual encoding of R, G and Band the 
reliability for the decoding is enhanced. 

(Fourth Embodiment) 

[0081 ] F1 G. 7 shows an example of a pattern structure 
for use in pattern light projection in a 3D image acquisi- 
tion method according to a fourth embodiment of the in- 
vention. 

[0082] The structure of the 3D image acquisition ap- 
paratus used in the fourth embodiment is substantially 
the same as that of the 3D image acquisition apparatus 
of the third embodiment. However, the form of the pat- 
tern used in the spatial pattern encoding method is dif- 
ferent. Specifically, as shown in FIG. 7, the pattern used 
in this embodiment is acolorstripe pattern in which local 
maximum/minimum positions of R, G and B signals are 
shifted. More specifically, the local maximum position of 
the R signal is displaced from the local maximum posi- 
tions of the remaining G and B signals. In other words, 
in the stripe pattern of this embodiment, the local max- 
imum luminance values and local minimum luminance 
values of each of the R, G and B component are alter- 
nately arranged, and the locaJ maximum position of the 
R signal coincides with the local minimum positions of 
the remaining G and B signals. 
[0083] In the pattern used in the 3D image acquisition 
in the fourth embodiment, the area with a local maximum 
luminance differs from the area with a local minimum 
luminance, and the local maximum positions and local 
minimum positions of R, G and B are shifted. In more 
general terms, in the pattern used in this embodiment, 
the areas with local maximum luminance values and the 
areas with local minimum luminance values of each of 
the color components are alternately arranged, and the 
position of the area with the local maximum luminance 
(the position of the area with the local minimum lumi- 
nance) of the pattern of at least one of the color compo- 
nents coincides with the position of the area with the 
local minimum luminance (the position of the area with 
the local maximum luminance) of the patterns of the oth- 
er color components. 

[0084] The method using this pattern is advantageous 
in that if the reflectances of all R, G and B signals are 
not at the noise level, the chroma become higher than 
in the case of the third embodiment in which the local 
maximum/minimum positions of all R, G and B are 
aligned, and the color determination process becomes 
easier than in the third embodiment. 



(Fifth Embodiment) 

[0085] The structure of a 3D image acquisition appa- 
ratus used in the fifth embodiment is substantially the 
5 same as that of the 3D image acquisition apparatus of 
the first embodiment. However, the form of the pattern 
used in the spatial pattern encoding method is different, 
as will be described below. 

[0086] FIG. 8 shows a pattern used in the 3D image 
10 acquisition method according to the fifth embodiment of 
the invention. This pattern structure is characterized in 
that the pattern is a gradation matrix pattern using gray 
scale values and local maximums and local minimums 
are alternately arranged in a two-dimensional fashion. 
'5 [0087] Since the local maximums and local minimums 
are alternately arranged in a two-dimensional fashion in 
the pattern used in the fifth embodiment, the extraction 
of a matrix image (boundaries of a block in a matrix) is 
advantageously facilitated, compared to the prior art. 
20 [Q088] According to this stripe pattern, encoding is ef- 
fected by using 2x2 blocks, instead of encoding using 
four successive numerals. Thus, the possibility of break- 
ing of a code is advantageously reduced in the case of 
measuring a vertically elongated object or a square ob- 
25 ject. 

(Sixth Embodiment) 

[0089] Although not shown, a sixth embodiment of the 
30 invention employs a matrix pattern in which blocks of 
the matrix, which have local maximum and minimum lu- 
minance values, are aligned among R, G and B. The 
structure of the 3D image acquisition apparatus may be 
substantially equivalent to that of the preceding embod- 
35 iment 

[0090] According to the sixth embodiment, since en- 
coding for R, G and B is individually performed, the de- 
coding process can be performed by combining the in- 
dividual encoding of R, G and B and the reliability for the 
40 decoding is enhanced. 

(Seventh Embodiment) 

[0091] A seventh embodiment of the invention em- 
45 ploys a matrix pattern (not shown) configured to have 
blocks of local maximum/minimum luminance values 
shifted among R, G and B. 

[0092] Similarly, the structure of the 3D image acqui- 
sition apparatus may be substantially equivalent to that 

so of the preceding embodiment. 

[0093] According to the seventh embodiment, the use 
of this matrix pattern is advantageous in that if the re- 
flectances of all R, G and B signals are not at the noise 
level, the chroma become higher than in the case where 

55 the blocks of local maximum/minimum luminance val- 
ues of all R, G and B are aligned, and the color deter- 
mination process becomes easier. 
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(Modification) 

[0094] The present invention may be modified as fol- 
lows. In the above-described embodiments, the lumi- 
nance (gradation) pattern adopts four gradations. From 5 
the standpoint of wide variety, it is preferable to provide 
at least three gradations. Besides, from the standpoint 
of easy recognition, it is preferable to provide six or less 
gradations, more preferably, five or less gradations. 
[0095] It is desirable that the entire pattern has the io 
feature that areas with local maximum luminance and 
areas with local minimum luminance are alternately ar- 
ranged. However, since comer portions are not actually 
used, there arises no practical problem if that part of the 
pattern, which is within the visual held of the imaging is 
optical system, has this feature. Considering the fact 
that the distance between the pattern projection means 
and the imaging means is sufficiently less than the dis- 
tance between the pattern projection means and the ob- 
ject, it should suffice if this feature is realized on that 20 
part of the projected pattern, which is projected within 
the range of the same solid angles as the imaging an- 
gles of the imaging means. 

[0096] In the case where the imaging angles are 1 0° 
in the vertical direction and 15° in the horizontal direc- 25 
tion, there is no practical problem if that part of the pro- 
jected pattern , which is projected within the range of 1 0° 
in the vertical direction and 15° in the horizontal direc- 
tion, has the feature that areas with local maximum lu- 
minance and areas with local minimum luminance are 30 
alternately arranged. 

[0097] Other modifications may be made without de- 
parting from the spirit of the invention. 
[0098] The embodiments of the invention have been 
described. The present specification includes the follow- 35 
ing inventions. 

(1) There is provided a spatial pattern encoding 
method wherein blocks or areas with local maxi- 
mum and minimum luminance values are alternate- *o 
fy arranged and a code is generated by combining 
local maximum/minimum information and gradation 
information. Since the local maximums and local 
mini mums are alternately arranged, extraction of a 
stripe image (stripe boundary) can be made easy. 45 

(2) There is provided a spatial pattern encoding 
method of item (1 ) wherein a stripe pattern or a ma- 
trix pattern is employed. Pattern projection images 
are obtained by imaging elements situated at loca- 
tions of two or more viewpoints. Not only the cone- so 
spondency between a projection-light pattern and a 
reception-light pattern but also the correspondency 
between plural images is performed. Thus, more 
exact correspondency is achieved and the meas- 
urement precision and reliability are enhanced. S5 

(3) There is provided a spatial pattern encoding 
method of item (1) wherein encoding is performed 
in substantially independent wavelength bands of, 



e.g. R, G and B. Since encoding for R, G and B is 
individually performed, the decoding process can 
be performed by combining the individual encoding 
of R, G and B and the reliability for the decoding is 
enhanced. 

(4) There is provided a spatial pattern encoding 
method of item (3) wherein blocks or areas with lo- 
cal maximum luminance values and local minimum 
luminance values of R, G and B are aligned. Since 
blocks with local maximum and minimum values are 
alternately arranged, the extraction of a matrix im- 
age (division of a block) can be made easier than 
in the prior art 

(5) There is provided a spatial pattern encoding 
method of item (3) wherein blocks or areas with lo- 
cal maximum luminance values and local minimum 
luminance values of R, G and B are shifted. As a 
result, if the reflectances of all R, G and B signals 
are not at the noise level, the chroma become high- 
er than in the case where the stripes of local maxi- 
mum/minimum luminance values of all R, G and B 
are aligned, and the color determination process 
becomes easier. 

(6) There is provided a 3D image acquisition appa- 
ratus comprising: 

a pattern projection section which projects a 
spatial encoding pattern of item (1); 
a pattern memory which stores the pattern that 
is projected by the pattern projection section; 
a light reception section which receives reflec- 
tion light; 

an image memory which stores information of 
the light received by the light reception section; 
a depth calculation section which calculates 
depth information of an object on the basis of 
the pattern stored in the image memory and the 
pattern stored in the pattern memory; 
a 2D image information generating section 
which generates 2D image information on the 
basis of the information stored in the image 
memory; and 

a 3D image generating section which gener- 
ates a 3D image on the basis of the 2D image 
information generated by the 2D image infor- 
mation generating section and the depth value 
calculated by the depth calculation section. 

As a result, since encoding for R, G and B is 
individually performed, the decoding process can 
be performed by combining the individual encoding 
of R, G and B and the reliability for the decoding is 
enhanced. 

(7) There is provided a 3D image acquisition appa- 
ratus according to item (6), comprising: 

a plurality of said light reception sections and a 
plurality of said image memories; 
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a correspondency determination section which 
matches the contents of the image memories 
and collates the images; 
the depth calculation section which calculates 
3D information of the object on the basis of in- 
formation determined by the correspondency 
determination section; and 
a 3D image generating section which gener- 
ates a 3D image on the basis of information 
generated by the 2D image information gener- 
ating section and the value calculated by the 
depth calculation section. 

[0099] According to the structure of item (7) of the 
present invention, if the reflectances of all R, G and B 
signals are not at the noise level, the chroma become 
higher than in the case where the blocks of local maxi- 
mum/minimum luminance values of all R, G and B are 
aligned, and the color determination process becomes 
easier. 

[0100] As has been described above, in the present 
invention, each band of R, G and B is encoded based 
on the combination of local maximum/minimum informa- 
tion and gradation information. The range of objects for 
3D shape measurement is not limited to white-based 
color objects and low-saturation color objects and is in- 
creased. 

[0101] Moreover, the present invention may provide 
a 3D image acquisition method for performing spatial 
pattern encoding and a 3D image acquisition apparatus 
for carrying out this method, wherein stripe boundary ex- 
traction from a stripe pattern projection image and block 
boundary extraction from a matrix pattern projection im- 
age can exactly be performed, and exact decoding is 
performed from an encoded projection image of an ob- 
ject even where the object is not of a white-based color 
or a low-saturation color. 



Claims 

1 . A 3D image acquisition apparatus comprising: 

a pattern projection section (102) which 
projects a predetermined pattern on an object; 
an imaging section (103, 104) which is dis- 
posed at a predetermined distance from said 
pattern projection section and images the ob- 
ject on which the predetermined pattern has 
been projected; and 

a depth calculation section (1 06) which detects 
the projection pattern projected on the object 
on the basis of an image acquired by said im- 
aging section, collates the detected projection 
pattern and the predetermined pattern project- 
ed by the pattern projection section, and calcu- 
lates a depth of respective parts of the object 
on the basis of the correspondency of the col- 



lation, 

characterized In that the predetermined pat- 
tern projected by said pattern projection section is 
s formed by alternately arranging areas with local 
maximum luminance values and areas with local 
minimum luminance values. 

2. A 3D image acquisition apparatus comprising: 

10 

a pattern projection section (102) which 
projects a predetermined pattern on an object; 
an imaging section (103, 104) which is dis- 
posed at a predetermined distance from said 
15 pattern projection section and images the ob- 

ject on which the predetermined pattern has 
been projected; and 

a depth calculation section (106) which detects 
an image pattern on the basts of an image ac- 

20 q uired by said imaging section , collates the de- 

tected image pattern and the predetermined 
pattern projected by the pattern projection sec- 
tion, and calculates a depth of respective parts 
of the object on the basis of the correspondency 

25 of the collation, 

characterized in that the predetermined pat- 
tern projected by said pattern projection section is 
formed by combining patterns of a plurality of color 
30 components, and the patterns of the plurality of 
color components are formed by alternately arrang- 
ing areas with local maximum luminance values and 
areas with local minimum luminance values. 

35 3. A 3D image acquisition apparatus according to 
claim 2, characterized in that in the patterns of the 
plurality of color components, positions of the areas 
with local maximum luminance values and positions 
of the areas with local minimum luminance values 

40 are aligned. 

4. A 3D image acquisition apparatus according to 
claim 2, characterized in that in the patterns of the 
plurality of color components, positions of the areas 

45 with local maximum luminance values and positions 
of the areas with local minimum luminance values 
are shifted. 

5. A 3D image acquisition apparatus according to 
so claim 2, characte ri zed In that the position of the 

area with the local maximum luminance of the pat- 
tern of at least one of the color components is 
aligned with the position of the area with the local 
minimum luminance of the patterns of the other 
55 color components. 

6. A 3D image acquisition apparatus according to 
claim 2, 3 or 4, characterized in that said plurality 
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of color components are RGB components. 

7. A 3D image acquisition apparatus according to any 
one of claims 1 to 4, characterized In that the pro- 
jected predetermined pattern is a stripe pattern or s 
a matrix pattern. 

8. A 3D image acquisition apparatus characterized 
by comprising: 

a pattern projection section (102) which 
projects on an object a spatial encoding pattern 
in which blocks or areas with local maximum 
and minimum luminance values are alternately 
arranged and encoding is effected by combin- 
ing local maximum/minimum information and 
gradation information; 

a pattern memory (101) which stores the pat- 
tern that is projected by the pattern projection 
section; 

a light reception section (104) which receives 
light reflected by the object; 
an image memory (105) which stores informa- 
tion of the light received by the light reception 
section; 

a depth calculation section (106) which calcu- 
lates depth information of the object on the ba- 
sis of the pattern stored in the image memory 
and the pattern stored in the pattern memory; 
a 2D image information generating section 
(1 07) which generates 2D image information on 
the basis of the information stored in the image 
memory; and 

a 3D image generating section (108) which 
generates a 3D image on the basis of the 2D 
image information generated by the 2D image 
information generating section and the depth 
information calculated by the depth calculation 
section. 

9. A 3D image acquisition apparatus according to 
claim 8, characterized in that said 3D image ac- 
quisition apparatus has a plurality of said fight re- 
ception sections (204R, 204L) and a plurality of said 
image memories (205R,2Q5L), and further includes 
a correspondency determination section (206) 
which matches the contents of the image memories 
and collates the images, 

the depth calculation section calculates 3D in- 
formation of the object on the basts of information 
determined by the correspondency determination 
section, and 

said 3D image section generates a 3D image 
on the basis of information generated by the 2D im- 
age information generating section and the value 
calculated by the depth calculation section. 

10. A 3D image acquisition method comprising: 



a step of projecting a predetermined pattern on 
an object; 

a step of imaging the object on which the pre- 
determined pattern has been projected, at a po- 
sition at a predetermined distance from a posi- 
tion where the predetermined pattern has been 
projected; and 

a step of detecting a pattern on the basis of an 
image acquired by said imaging step, collating 
the detected pattern and the pattern projected 
by said pattern projection step, and calculating 
a depth of respective parts of the object on the 
basis of the correspondency of the collation, 

characterized In that the predetermined pat- 
tern projected by said pattern projection step is 
formed by alternately arranging areas with local 
maximum luminance values and areas with local 
minimum luminance values. 

11. A 3D image acquisition method comprising: 

a step of projecting a predetermined pattern on 
an object; 

a step of imaging the object on which the pre- 
determined pattern has been projected, at a po- 
sition at a predetermined distance from a posi- 
tion where the predetermined pattern has been 
projected; and 

a step of detecting a pattern on the basis of an 
image acquired by said imaging step, collating 
the detected pattern and the pattern projected 
by said pattern projection step, and calculating 
a depth of respective parts of the object on the 
basis of the correspondency of the collation, 

characterized in that the predetermined pat- 
tern projected by said pattern projection step is 
formed by combining patterns of a plurality of color 
components, and the patterns of the plurality of 
color components are formed by alternately arrang- 
ing areas with local maximum luminance values and 
areas with local minimum luminance values. 

12. A 3D image acquisition method, to which applied is 
a spatial pattern encoding method wherein blocks 
or areas with local maximum and minimum lumi- 
nance values are alternately arranged and a prede- 
termined code pattern is generated by combining 
local maximum/minimum information and gradation 
information. 

13. A 3D image acquisition method according to claim 
1 2, characterized In that the code pattern obtained 
by the spatial pattern encoding method is a stripe 
pattern or a matrix pattern. 

14. A 3D image acquisition method according to claim 
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1 2, characterized in that the code pattern obtained 
by the spatial pattern encoding method is encoded 
in substantially independent wavelength bands 
such as R, G and B. 

15. A 3D image acquisition method according to claim 
1 4, characterized In that blocks or areas with local 
maximum luminance values and local minimum lu- 
minance values of R, G and B are aligned. 

16. A 3D image acquisition method according to claim 
1 4, characterized in that blocks or areas with local 
maximum luminance values and local minimum lu- 
minance values of R, G and B are shifted. 
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